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1.  INTRODUCTION1 

Our  objective  is  to  provide  a  more  general  theoretical  basis  for 
those  methods  for  solving  constrained  minimization  problems  that  are 
based  on  successive  unconstrained  minimizations  of  a  parametric 
auxiliary  function.  In  order  to  do  this,  we  first  give  basic  defining 
properties  of  a  general  auxiliary  function  and  obtain  a  proof  of  local 
convergence  for  the  mildly  regulated  nonconvex  problem.  We  proceed 
from  this  general  result  to  develop  a  family  of  auxiliary  functions  by 
giving  the  general  function  more  and  more  structure,  eventually  being 
led  to  the  general  form  of  the  "  penalty"  functions  that  conventionally 
have  been  utilized.  Similarly,  the  problem  structure  is  increasingly 
specialized  and  we  deal  finally  with  the  convex  problem.  As  expected, 
stronger  results  are  obtained  as  additional  structure  is  assumed. 


1]  This  paper  Is  based  on  material  developed  in  [1]  and  essentially  constitutes 
a  generalization  of  results  originally  obtained  in  [5], 
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The  problem  of  interest  is  the  general  mathematical 
programming  problem 

(A)  minimize  f(x) 

subject  to  g.(x)  5  0  ,  i=l,2,...,p  and 

hj  (x)  =  0  ,  j=l,  2, ...»  q  ,  where  x  e  En. 


f 


Since  we  wish  to  assume  little  more  than  the  continuity  of 
the  problem  functions  for  many  of  our  results,  we  must  account  for 
local  minima  in  problem  (A).  Even  with  local  minima  present,  we 
could  still  take  the  point  of  view  that  only  global  minima  are  of 
interest,  and  concentrate  on  analyzing  what  is  needed  to  bring  about 
convergence  to  a  global  solution.  However,  our  point  of  view  is  to 
attribute  importance  to  the  determination  of  any  local  minimum 
objective  function  value  v*  of  problem  (A),  and  to  deterxr.ine  at  least 
one  local  minimum  x*  such  that  f(x*)  =  v*.  Global  results  follow 

as  a  byproduct  of  local  results  in  this  approach. 

In  view  of  this  objective,  we  must  guarantee  that  a  sequence 
of  points  will  be  generated  converging  to  any  set  of  local  minima 
yielding  a  given  objective  function  value.  For  the  techniques  that  we 
shall  develop  here,  a  key  j  equirement  is  that,  in  the  interior  of  a 
specified  domain  of  interest,  each  such  set  contains  a  suitably  prescribed 
closed  subset.  In  the  development  given  in  [5]  ,  it  is  assumed  that  any 
such  set  is  compact.  V.Te  shall  utilize  this  assumption,  concentrating 
on  generating  bounded  sequences  converging  to  finite  local  solutions  of  (A). 

V.'e  address  ourselves  to  developing  methods  that  generate 
a  sequence  of  points  converging  to  a  local  solution  of  (A)  by  means  of 
successive  unconstrained  minimizations  cf  a  parametrized  auxiliary 
function,  over  a  specified  sequence  of  values  of  the  involved  parameters. 
Also,  we  restrict  our  attention  to  auxiliary  functions  which,  under 
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suitable  conditions,  converge  in  value  to  the  value  of  the  objective 
function  f(x)  at  a  local  minirrum  of  (A).  In  our  development,  we  are 
less  concerned  with  the  forrr.  of  the  penalty  function  and  concentrate  on 
those  properties  the  function  must  possess  in  order  to  define  a  con¬ 
vergent  sequential  unconstrained  minimization  scheme.  Our  results  are 
considerably  more  general  than  those  obtained  previously. 

Particular  realizations  of  sequential  unconstrained  methods 
in  the  class  we  shall  develop  have  been  only  recently  validated,  by 
Paricot  [7]for  the  linear  problem,  by  Pietrzykowski  TO],  Fiacco  and 
IwcCorrr.ick  [2]  ,  [3]  ,  [4]  ,  and  Pnmentale  [9]  for  the  convex  problem., 
and  by  Stong  [10]  ,  Zangwill  [12]  and  Fiacco  and  h.cCormick  T5]  for 
nonconvex  problems  as  well.  Iviany  of  these  results,  and  in  particular 
the  principal  convergence  theorems,  will  be  largely  subsumed  in  the 
development  following. 

In  Section  2,  we  define  a  generalized  auxiliary  function  ard 
prove  that  it  yields  a  sequence  of  local  unconstrained  minima  whose 
limit  points  are  local  solutions  of  the  problem,  under  mild  regulatory 
conditions.  The  function  is  defined  in  such  a  manner  that,  if  the  set 
of  points  satisfying  a  specified  constraint  has  a  nonempty  interior,  then 
the  constraint  can  be  enforced  for  all  points  in  the  minimizing  sequence. 

In  Section  3,  this  function  is  somewhat  specialized  to  yield 
the  general  forms  of  "interior-feasible"  and  "exterior-feasible" 
auxiliary  functions.  The  convergence  theorems  for  these  functions 
follow  as  immediate  corollaries  of  the  above  general  result. 

Further  specializations  lead  to  the  functions  defined  and 
developed  in  [5] .  These  subsume  the  class  of  interior  point  and 
exterior  point  "penalty"  functions  that  have  conventionally  been  utilized 
in  this  approach.  A  stronger  characterization  of  convergence  is  given 
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for  these  functions,  in  Section  4. 

For  the  convex  programming  problem,  the  interior  and 
exterior  penalty  functions  can  be  defined  to  be  convex,  thereby 
leading  to  global  and  dual  results.  These  were  developed  in  [5] 
and  are  summarized  in  Section  5. 


1 
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.  The  Generalized  Auxiliary  Function  for  the  Mixed  Algorithm 

For  convenience,  we  shall  write  problem  (A)  in  the  form 
(B)  minimize  f(x)  subject  to  g^(x)  2  0  ,  i  =  l,2,...,q. 

The  following  definitions  and  results  will  be  used.  The  first 
is  a  trivial  extension  of  the  fact  that  a  continuous  function  attains 
its  minimum  on  a  compact  set. 

Lemma  2.  L 

If  F(x)  is  continuous  in  the  nonempty  interior  of  a  compact  set 
V  and  such  that  possibly  F(y)  =  +  °°  for  y  c  (V-  V°),  then  there 
exists  a  finite  value  v  and  a  point  X  e  y  such  that  F(~ )  =  y  s  min 


vF(x). 


Definition. 

A  point  x*  C  G  *  {x  Jg^(x)  5  0,  i  =  1, . . .  ,  q]  is  a  local  minimum  of 
problem  (B),  if  there  exists  an  open  set  N  such  that  x*  e  N  and 
f(x*)  =  min^^^fx).  If  N  can  be  selected  such  that  f(x+)  =  min^f(x), 
x*  is  referred  to  as  an  unconstrained  local  minimum  of  (B). 
Otherwise,  x*  is  called  a  constrained  local  minimum  of  (B). 

The  lemma  following  will  be  used  in  proving  the  convergence  of 
a  minimizing  sequence  to  a  prescribed  compact  set  of  minima 
yielding  a  given  local  minimum  value  of  f(x)  in  problem  (B). 

For  significant  additional  generality,  we  require  the  definition 
of  an  "isolated"  set. 

Definition. 

A  nonempty  set  A*  c  A  is  called  an  isolated  set  of  A  if  there 
exists  a  closed  set  E  such  that  E°  3  A*  and  such  that  if 
x  e  (E-A*),  then  x  4  A. 
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Y/c  assume  continuity  of  the  functions  of  problem  (B)  and 
consider  as  isolated  set  A*  of  local  minima  A  yielding  a  particular 
local  minimum  value,  say  f(x)  =  v*.  The  following  result  asserts 
that  there  exists  a  compact  set  S  such  that  S°  =>  A*  ,  and  such  that 
the  set  of  global  minima  of 

(B*)  minimize  f{x)  subject  to  x  cGf]  S 

is  given  by  A*  . 

Let  A  =  (x!f(x)  =  v*  and  x  is  a  local  solution  of  (B)}. 

Lemma  2.2.  Existence  of  Compact  Perturbation  Set. 

If  the  functions  of  (B',  are  continuous  and  if  A*  is  a 
nonempty  compact  isolated  set  of  A,  then  there  exists  a 
compact  set  S  such  that  5°  3  A*  and,  if  xc  Gf|S  and 
minG^sf(x)  =  f (x)  ,  then  x  cA*  . 

Proof.  B/  assumption  there  exists  a  closed  set  E  such  that 
E°  3  A*  and  xe  (E-A*)  implies  x  j  A  .  Since  A*  is  compact,  it 
follows  that  there  exist  compact  sets  such  that  A*  cs”cE  and 

such  that  [S^}J,A*  .  If  the  conclusion  of  the  lemma  is  false,  then 

k  k  k  x 

min^^g  f(x)  =  f(x  )  5  v*  ,  with  x  cGOS^  and  x  $  A*  for  every  k. 

k 

Since  A*  is  closed  and  since  {x  }  must  contain  a  convergent  sub- 

k.  k.  _ 

sequence  [x  ’}  ,  it  follows  that  x  J  -xe  A*  . 

k.  _  _ 

If  f(x  J )  <  v*  =  f(x)  for  all  j  ,  the  above  implies  that  x  is  not 

a  local  minimum  of  (B),  a  contradiction  of  the  definition  of  A*.  If 
k.  _ 

f(x  J  •)  =  v*  for  some  j  =  j  then,  by  construction  of  {S^l  ,  v*  must  be 
the  minimum  value  of  f(x)  in  G  D  for  j  -  j  .  But  since  x^  e  s£_ 

k  3  j 

for  k  large  enough,  this  means  x)  is  a  local  solution  of  (B)  with 
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value  v*  for  j  larj'e  enough;  i.  e. ,  x-^e  A/^E  and  hence,  x  eA* 
for  all  j  large  enough.  Thus,  the  conclusion  follows  for  this  case,  also. 

Q.  E.  D. 

The  equivalent  problems  (A)  and  (B)  can  also  be  formulated  as 
(M)  minimize  f(x)  s.  t.  xeRf^Ci 

where  R  =  {  x  |g.(x)  £  0,  i  =  1, . . .  ,m}  and  Q  =  (x  |g.(x)  £  0,  j=m+l, . . . ,  q}  . 
If  there  are  any  equality  constraints,  we  assume  these  appear  among  the 
last  q  -  m  constraints  (i.  e.  ,  they  are  involved  only  in  defining  C*  and 
not  in  defining  R).  However,  Q  may  contain  inequality  constraints  as 
well. 

It  is  assumed  that  R°  and  O  f  .  It  is  desired  to  solve 
(M)  by  generating  a  sequence  of  unconstrained  minima  of  an  auxiliary 
function  in  such  a  manner  that  the  sequence  must  be  restricted  to  R°  , 
but  need  not  be  restricted  to  C  . 

As  throughout,  we  tacitly  assume  that  x  e  En  and  the  functions 
of  problem  (M)  are  continuous.  The  generalized  auxiliary  function  is 
defined  as  follows. 

Defining  Properties  of  V(x,  r,t). 

i)  V(x,  r,t)  is  continuous  for  x  e  R°  ,  for  any  r  >0  and  t  >  0; 

ii)  If  {x^}  c  R°  and  x^  -* y  c  R  -  R°,  where  Jl y JJ  <  “,  then 

v(xk,  r,  t)  =  +»  for  any  r  >  0,  t  >  0; 

iii)  If  {x^l  c  R°  ,  r^  >  0  and  t^  >  0  for  every  k,  and  (x^,  r^»  t^)  -♦ 

-*  (y»  0,  +“),  with  || y  jj  <  ®,  then  lim  im^  V(x^,  r^,  t^)  r*=+®ify^Q, 

/  S  f(y)  otherwise; 
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iv)  If  y  eR ° C\  Q,  >  0  and  t^  >  0  for  every  k,  and  (r^,  t^)  -*  (0,  +®), 
then  limk-aV(y)  rk,  tR)  =  f(y)  . 

Realizations  of  V(x,  r,t  )  follow  in  the  ensuing  development. 
With  V(x,  r,t)  so  defined,  a  sequential  technique  having  the 
desired  properties  can  be  validated  under  suitable  conditions.  Let 
A  =  {xjf(x)  =  v*  and  x  is  a  local  solution  of  (M)}. 

Theorem  2.1.  Convergence  of  V-minima  to  Local  Solutions  of  (M). 


If  the  functions  of  (M)  are  continuous,  V(x,  r,t)  is  as  above, 

R°  (~\  Q  ^  ^  »  R  C  =  C  »  A*  is  a  nonempty  compact 

isolated  set  of  A,  r^  >  0  and  t  >  0  for  every  k  with 

(r^,t^)  -  (0,  +<»)  ,  then:  (a)  there  exists  a  compact  set  S  as 

given  in  Lemma  2.  2,  so  that  S°C5  A*  and,  for  all  k  large 

k 

enough,  the  unconstrained  minima  [x  }  of  {V(x,r^,t^)j  in 
R  /IS  exist  and  every  limit  point  of  the  uniformly  bounded 
sequence  {x^}  is  in  A*,  and  (b)  V(x^,  r^,  t^)  -♦  v*  . 

Proof.  Let  V(x  ,  r^,  t^)  =  inf^  ^  g  V(x,  r^,  t^)  ,  where  S  is  the 
compact  set  defined  in  Lemma  2.  2.  Recall  that  this  also  means  that 
R  0  S°  3 A*.  Since  A*  C  R  0  Q  and  R  P\  P  =  R°f)  Q  ,  it  follows  that 

R°fi  s  H . 

By  property  (i),  V(x,  r,t )  is  continuous  ir  R  f]  S  for  r  >  0 

and  t  >  0.  By  (ii),  V(x,  r,  t)  -♦  +®  as  x  -•  y  if  x  e  R°  and  y  e  (R-R°), 

for  any  r  >0,  t  >  0.  These  facts  imply  that  [x  }  c  R  f|  S.  The  fact 

k 

that,  for  any  r^  >0,  t^  >0,  x  exists  minimizing  V(x,  r^#^)  in  the 
compact  set  R  f|  S  follows  from  Lemma  2. 1. 

Since  R  f|  S  is  compact,  (x  )  mus1:  have  a  convergent 

k 

subsequence  which,  for  convenience,  shall  still  >e  denoted  by  [x  ] . 
Hence,  we  can  assume  x^  -*y°eR  D  S.  In  particular,  this  also  means 


o 


| yu  j)  <  oo.  V'e  shall  show  that  y  €  A*  . 

By  hypothesis,  R  Q  is  the  closure  of  R°Q  Q  and  hence,  by 
the  continuity  of  f(x),  there  exists  x°  e  R°/^|  Q  f)  S  such  that 

O 

f(x  )  S  v*  +  e  ,  any  e  >  0  .  By  definition  of  x  , 


(2. 1)  V(x  ,  rk,  tk)  S  V(x  ,  rk,  tk) 


and  now  property  (iv)  implies  that 


(2.  2)  lim  infk  V(xk,  rk> tk)  -  lim  infk  V(x°,  rk» tk)  =  f(x°)  <  v*  +  e 


for  any  e  >0  ,  by  suitable  choice  of  x  . 

o  k 

If  y  ^  C  then,  by  property  (iii),  lim^  Vfx  ,  rk»tk)  =  +®, 

which  contradicts  (2.  2).  Thus,  y°  €  R  0  S  D  Q  . 

If  y°  ^  A*  then,  by  definition  of  S  ,  it  follows  that  f(y°)  ^  v*  +  X  , 

for  some  X.  >  0  .  By  property  (iii)  this  means  that 
k.  o 

lim  infk  V(x  ,  rk»  tk)  -  f(y  )  2  v*  +  X  ,  which  again  contradicts  (2.  2),  since 

we  can  select  x°  above  such  that  e  <  X  .  Consequently,  we  must  have 

o  .  % 
y  e  A  . 

Further,  since  [xk]  C  R°  ,  xk  -•  y°  and  y°  e  A*  crD  S°  ,  this 
also  means  that  for  k  large  enough,  xk  cR0/]  S°  .  Thus,  the  xk  are 
uniformly  bounded  unconstrained  minima  of  V(x,  rk<  tk)  in  R°0  S°  for 
large  k,  and  every  limit  point  of  [x  }  is  in  A*. 


For  part  (b  ),  since  y°  eA*  ,  (2.2)  and  property  (iii)  imply  that 


(2.  3)  v*  =  f(y°)  <  lim  infk  V(xk,  rR,  tfc)  <  v*  +  e  , 


for  any  e  >  0  .  Similarly,  (2. 1)  and  properties  (iii)  and  (iv)  yield 


% 


(2.4)  v*  S  lim  sup^^  V(xk,  rk,tk)  S  v*  +  e  , 

for  any  e  >0  .  From  (2.  3)  and  (2.  4),  lim^^  V(xk,  r k,  tk)  =  v*  . 

O.  E.  D. 

The  following  corollary  gives  the  global  result  indicated  in  the 
previous  discussion.  It  will  be  apparent  that  this  result  follows  immediately 
from  the  theorem  and  the  fact  that  the  set  of  global  minima  of  (M)  must  be 
closed  under  our  assumptions  on  the  problem  functions.  The  second 
corollary  is  an  obvious  consequence. 

Corollary  1.  Convergence  to  Bounded  Set  of  Global  Minima. 

Assuming  the  hypotheses  of  Theorem  2.  1,  if  the  set  A*  of  global 
minima  of  problem  (M)  is  bounded,  then  the  conclusions  of  the 
theorem  hold  and  the  sequence  of  unconstrained  local  minima  {xk} 
of  {V(x,  rk>tk)}  in  P.  C\  S  is  such  that  every  limit  point  is  a  global 
minimum  of  (M). 

Corollary  2.  Convergence  to  Isolated  Local  Minimum. 

Assuming  the  hypotheses  of  Theorem  2.  1,  if  A*  =  {x*}  and  x*  is  an 
isolated  local  minimum  of  (M),  then  the  conclusions  of  the  theorem 
hold  and,  furthermore,  the  sequence  of  unconstrained  local  minima 
{xk}  of  {V(x,rk>tk)}  in  P.  flS  is  such  that  {x*}  itself 
converges  to  x*. 

Before  proceeding  with  the  general  development,  a  few  interesting 
facts  may  be  noted  about  the  general  auxiliary  function  that  has  been 
defined.  Suppose  this  function  has  the  form  V[f(x),  g^(x), .  . .  ,  g^(x),  r,  t]  , 

V  is  once  differentiable  in  f,  g.,...  ,g  ,  and  these  latter  functions  are 

•  o  k 

differentiable  in  x,  for  x  eR  and  any  r  >0,  t  >0  .  Then,  if  x  is  a 


local  unconstrained  minimum  of  V,  denoting  the  corresponding  value  of  V 


j 


by  V^,  it  follows  that 


_,rk  SV  ,  2  dV  „  ,  k4  „ 

VV  =  — r  Vf  (x  )  +  £  - j—  7g  (x  )  =  0  . 

9f(x  )  1-1  3g.(x  ) 

k  ic 

This  equation  is  the  same  as  VL  (x  ,u  )  =  0  ,  where  L  is  the  Lagrangian 
associated  with  Problem  (M),  providing  we  assume  dV/dfCx^)  ^  0  and  we 


k  -3V/3  gj(xK) 

i  =  - r - 

av/af(xK) 


This  shows  how  we  can  be  led  to  establishing  a  direct  connection  between 

the  conditions  that  hold  at  a  local  minimum  of  the  penalty  function  and 

the  conditions  that  hold  at  a  local  solution  of  (M). 

The  above  also  indicates  the  connection  with  duality  in  the  convex 

programming  problem  that  will  be  briefly  summarized  in  Section  5.  If 

V  is  convex  increasing  in  the  convex  function  f  and  decreasing  in  the 

concave  functions  g^(x),  i  =  1, .  .  .  ,  q,  then  it  follows  that  V(x,  r,t)  is 

a  convex  function  in  R°  .  If  we  further  assume  3V/^f  >  0  ,  then  it  follows 

k  k 

immediately  that  (x  ,u  )  is  a  dual  feasible  point. 

Thus,  the  connections  between  the  conditions  associated  with 
minimization  of  the  penalty  function  and  the  optimality  and  duality 
conditions  are  apparent  simply  from  the  structure  of  this  function,  our 
defining  properties  having  assumed  very  little  concerning  its  particular 


form. 
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3.  Generalized  Interior  and  Exterior  Auxiliary  Functions 

Returning  to  the  general  development,  we  wish  to  be  able  to  solve 
Problem  (M)  when  we  require  only  that  x  e  R,  or  that  x  e  Q,  i.  e.  ,  we 
wish  to  solve  the  problems 


M(R) 

minimize  f(x) 

subject  to  x  e  R, 

and 

M(Q) 

minimize  f(x) 

subject  to  x  e  0  , 

using  an  auxiliary  function  technique  analagous  to  that  given  above  for 
Problem  (M). 

The  Problems  M(R)  and  M(Q)  have  the  same  basic  structure,  the 
essential  difference  being  that  we  shall  assume  R°  ^  and  Q  ^  ^  (so 
that  we  may  have  C°  =  <}).  The  remaining  distinction  is  procedural:  we 
insist  on  restricting  x  to  R°  in  the  course  of  solving  M.(R),  whereas 
x  need  not  be  restricted  to  Q  in  the  course  of  solving  M(G)  . 

Note  that  M  is  the  same  as  M(R)  if  Q  =  En  and  M  is  the  same  as 
M(C)  if  R  =  En.  Since  there  ere  no  restrictions  on  R  and  Q  in  the 
above  development  (other  than  R°  4  ,  Q  f  (|  ),  the  above  convergence 

theorem  for  Problem  (M)  utilizing  V(x,  r,  t)  is  valid  for  R  =  E°  or 
G  =  En  .  V'e  shall  modify  the  respective  definitions  of  V(x,  r,t)  in  that 
we  shall  associate  the  parameter  r  with  Problem  M(R)  and  the 
parameter  t  with  Problem  M(Q)  . 

V’e  are  thus  led  to  the  auxiliary  functions  for  M(R)  and  M(Q)  which 
we  shall  call  U(x,  r)  and  T(x,t),  respectively. 

We  assume  R°  f  (j)  and  essentially  arrive  at  the  defining  properties 
of  U(x,  r)  by  setting  Q  =  En  and  by  suppressing  t  in  the  definition  of 
V(x, r, t)  . 


Defining  Properties  of  U(x,  r). 


a)  U(x,  r)  is  continuous  for  x  e  R°,  for  any  r  >  0; 

b)  If  {xk}  C  R°  and  x^  ->  y  £  R  -  R°,  where  '•  y <®,  then 
lin.^^  U(x  ,  r)  =  +°°  for  any  r  >  0; 

c)  If  {xk]cR°,  r^  >  0  for  every  k,  and  (xk,  r^)  -*  (y,  0),  where 
If y  |j  <oa*  ^en  lim  infk  U(xk,  r(<)  *  f(y)  ; 

d)  If  y  e  R°,  r,  >  0  for  every  k,  and  r  -•  0,  then 

K  K 

limk-.®u(y'rk)  =  f(y)  • 

Assuming  C  f  d  ,  setting  R  =  En  and  suppressing  r  in  the 
definition  of  V(x,  r,t),  we  obtain  the  defining  properties  of  T(x,t). 

Defining  Properties  of  T(x,  t). 

A)  T(x,t)  is  continuous  for  x  cEn,  for  any  t  >  0; 

E)  If  {xk}  c  En,  t^  >  0  for  every  k,  and  (xk, t^)  -*(y, +°°)>  where 
!,y[|  <  00 ,  then  lim  inf^  T(xk,tk)  f  =  +®if  y  ^  Q, 

I  >  f (y)  otherwise; 

C)  If  y  c  C,  t^  >  0  for  every  k,  and  t^-^00,  then 

limk-»T^y«tk)  =  f(y)  • 

As  direct  consequences  of  these  definitions  and  the  previous  results, 
we  obtain  methods  for  solving  M(R)  and  M(C)  • 

Theorem  3.  1.  Convergence  of  U-Minima  to  Local  Solutions  of  M(R). 

The  convergence  theorem  for  V(x,  r,t)  applied  to  (M)  is  valid  if 
V(x,  r,t)  is  replaced  by  U(x,  r),  (M)  is  replaced  by  M(R),  C  =  En, 
and  t  is  suppressed  ,  in  the  statement  of  that  theorem. 

Theorem  3.  2.  Convergence  of  T-Kainima  to  Local  Solutions  of  M{Q). 

The  convergence  theorem  for  V(x,  r.  t)  applied  to  (M)  is  valid 
if  V(x,  r,t)  is  replaced  by  T(x,  t),  (M)  is  replaced  by  M(Q), 

R  =  En,  and  r  is  suppressed,  in  the  statement  of  that  theorem. 
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The  steps  in  the  proofs  of  these  two  theorems  are  precisely  the 
same  as  those  in  the  V-function  theorem,  with  obvious  modifications,  on 
making  the  indicated  substitutions. 

4.  Additional  Results  for  General  Interior  and  Exterior  Penalty  FnnUions 
We  obtain  a  realization  of  V(x,  r,t)  in  terms  of  U(x,  r)  and 
T(x,  t)  by  defining  V^(x,  r,t)  =  U(x,  r)  +  T(x,t)  -  f(x).  As  required  by  the 
definition  of  V(x,  r,t),  we  assume  the  problem  functions  continuous, 

R°  ,  and  Q  ^  (jl  .  The  following  result  is  a  direct  consequence  rf 
the  defining  properties. 

Lemma  4.  L  Y^(x,  r,  t)  is  a  V-function  (i.  e.  ,  a  function  V(x,  r,  t)  satisfying 
(i)  through  (iv)  above.  ) 

"Penalty"  functions  used  in  this  approach  are  usually  obtained  by 
adding  to  f(x)  a  "penalty  term"  which  absorbs  the  effects  of  the  constraints 
of  the  given  problem  and  the  involved  parameter.  Towards  developing 
these  functions  as  particular  realizations  of  the  functions  U(x,  r)  and 
T(x,t)  defined  above,  we  define  the  following.  Assume  R°  ^  (jl  and 

Q  H  • 


Defining  Properties  of  I(x,  r). 

a^)  I(x,  r)  is  continuous  for  x  sR°,  for  any  r  >0; 

b^)  If  {x^}  c  R°  and  x‘C  -  y  f:  R  -  R°,  where  j| y  *j  <  00 ,  then 

lim^_t(jjI(x  ,  r)  =  +°°  for  any  r  >0  ; 

k  o  k 

Cj)  If  [x  }  c  R  ,  r^  >0  for  every  k,  and  (x  ,  r^)  -  (y,  0),  where 

k 

y  j,  <  ®,  then  lim  inf^  I(x  ,  r^)  £  0; 
d.)  If  y  eR°,  r,  >  0  for  every  k,  and  r  -•  0,  then 

IK  K 

lin-k..=oI<y'rk>  =  °- 

Defining  Properties  ef  0(x,t). 

A^)  0(x,t)  continuous  for  x  En,  for  any  t  >  0; 


\ 
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Ej)  If  {xk}  c  En,  >0  for  every  k,  and  (x\  t^)  -  (y,+co),  where 
'|y  <  ®,  then  lirr.  inf^  0(x^,  t^)  C-  +“  if  y  ^  Q 


0  otherwise; 

Cj)  If  y  €  O,  t^  >  0  for  every  k,  and  t^  -•  +00,  then 

lim.  0(y,  t.  )  =  0. 
k-“  7  k 

With  "U-function"  and  ,rT-function"  meaning  U(x,  r)  and  T(x,t), 
respectively,  as  defined  in  the  previous  section,  we  obtain  the  following 
direct  consequences. 

Lemma  4.  2.  U^(x,  r)  s  f(x)  +  I(x,  r)  is  a  U-function. 

Lemma  r.  3.  T^(x,  t)  =f(x)  ■  0(x,  t)  is  a  T-function. 

The  proofs  of  the  lemmas  follow  immediately  from  the  continuity 
of  f(x)  and  the  defining  properties. 

A  further  immediate  consequence  of  the  three  lemmas  above  is  the 
following. 

Corollary  1.  V^x,  r,t)s  U^(x,  r)  +  T^(x,  t)  -  f(x)  =  f(x)  +  I(x,  r)  +  0(x,  t) 
is  a  V-function. 

This  provides  a  realization  of  the  V-function,  associated  with  Problem 
(M),  in  terms  of  the  objective  function  f(x)  of  that  problem,  and  the 
penalty  functions  associated  with  problems  M(R)  and  Li(Q)  (through 
Uj(x,r)  and  T^(x,t)  above). 

The  penalty  functions  defined  in  [5]  for  the  problems  M(R), 

M(O),  and  (Ivi)  are,  respectively, 

U2(x,  r)  =  f(x)  +  s(r)  I(x)  , 

T2(x,  t)  s  f(x)  +  p(t)  0(x)  , 

and  V2(x,  r,t)  =  f(x)  +  s(r)  I(x)  +  p(t)  0(x)  . 

The  penalty  functions  utilized  to  date  generally  subscribe  to  the 


above  forms.  We  shall  summarize  the  defining  properties  of 


“  Li  ■ 


8(r),  I(x),  p(t),  and  0(x).  It  follows  readily  that  s(r)  I(x)  is  an  I-function 
(satisfies  properties  (a^)  through  (d^)  above),  and  p(t)  0(x)  is  an  0-function 
(satisfies  (A^)  through  (C^)  above). 

Defining  Properties  of  I(x)  and  s(r)  [5], 

1)  I(x)  is  continuous  for  x  e  R°; 

2)  If  {xk}  c  R°  and  x^  -  y  s  R  -  R°,  then  lim^^^^x^)  =  ; 

3)  s(r)  is  a  (scalar- valued)  function  of  r,  continuous  for  r  >0; 

4)  If  r^  >  r^  >0,  then  s(r^)  >  s(r2)  >  0; 

5)  If  r.  >  0  for  every  k  and  r.  -*  0,  then  lim  s(r  )  =  0. 

K  K  IT  i  K 

k-*oo 

Defining  Properties  of  0(x)  and  p(t)  [5]. 

1  n 

1  )  0(x)  continuous  for  x  e  E  ; 

21)  0(x)  C  =  0  if  x  c  C  , 

)  >0ifx*C; 

1 

3  p(t)  is  a  (scalar-valued)  function  of  t,  continuous  for  c  >  0; 

41)  If  t2  >  tj  >  0,  then  p(t2)  >  p  (tj)  >  0; 

5*)  If  t^  >  0  for  every  !  ind  t^  -*+® ,  then  lin  ^^ptt^)  =  +®. 

As  straightforward  consequences  of  these  defining  properties  we 
obtain  the  following: 

Lemma  4.  4.  I^(x,  r)  =  s(r)  I(x)  is  an  I-function. 

Lemma  4.  5.  0^(x,t)  =p(t)  0(x)  is  an  0-function. 

In  view  of  the  above  results,  this  means  that  is  a  U-function 
(1.  e.  ,  satisfies  (a)  through  (d)  above),  is  a  T-function  (i.  e.  ,  satisfies 
(A)  through  (C)  above)  and  is  a  V-function  (i.  e.  ,  satisfies  (i)  through 
(iv)  above.  ) 

From  this  point  on,  we  shall  be  dealing  only  with  the  U^,  and 
functions  defined  above.  For  convenience,  therefore,  we  shall  revert 
to  the  notation,  U,  T,  and  V  for  these  functions. 


*'  iU  “ 


It  is  first  noted  from  the  above  development  that  these  penalty 
functions  are  special  cases  of  the  general  auxiliary  functions  defined  in 
Sections  2  and  3,  so  that  the  respective  convergence  theorems  apply. 
Additional  characterizations  of  convergence  follow  as  corollaries  of  these 
theorems,  from  the  particular  structure  ©f  these  functions. 

In  the  following  we  assume  the  conditions  of  Theorems  3.  1  and  3.  2 
are  satisfied. 

V.  e  know  [xk]  cR°fl  S°  for  k  large,  where 

1c 

minRp\  S  U(x’  rk*  =  U*x  ’  rk*  1 

and  S  is  the  compact  set  defined  in  Lemma  2.  2.  Recall  that  S°  3A*,  a 
compact  set  of  local  minima  associated  with  the  value  f(x)  =  v*.  Also, 
we  know  there  exists  at  least  one  limit  point  x*  of  {xk}  ,  and  any  such 
limit  point  is  in  A  .  The  following  results  show  that  for  U(x,r)  as 
presently  defined,  we  can  assure  that  the  minimizing  points  yield  values 
of  f,  U  and  1  that  converge  monotonically. 

Corollary  1  of  Theorem  3.  1. 

If  U(x,  r)  =  f(x)  +  s(r)  I(x),  where  I{x)  and  s(r)  satisfy  properties 
(1)  “  (5)  given  above,  f(x)  is  continuous,  is  a  positive  strictly 

decreasing  null  sequence,  and  min^  ^  g  U(x,  r^)  =  U(x  ,  r^)  , 
then  f(xk)  v*  ,  I(xk)  f  r> ,  where  'y  =  +»  if  x*  €  (R-R°)  and 
|?.'J  <  00  if  x*  e  R°,  and  s(rk)  I(xk)  -«  0.  Also,  if  I(x)  2  0  in 

R°n  S  ,  then  U(xk,  Tfc)  |  v*  . 

_ 

Proof.  By  definition  ef  x  ,  U(x  ,  r^)  ^  U(x,  r^)  for  every 
x  e  S  f)  R,  so  that 

(4.  1)  f(xk)  +  s(rk)  l(xk)  <  f(xk+1)  +  s(rk)  I(xk+1),  and 

(4.  2)  f(xk+1)  +  s(rk+1)  I(xk+1)  <  f(xk)  +  s(rR+1)  I(xk). 

Multiplying  the  first  inequality  by  s(rk+^),  the  second  by  s(rk), 


-  J  /  • 


summing  and  rearranging  yields 

[s(rk)  -  s(r|_+1)]f(xk+1)  <  [s(rk)  -  s(r  k+1)]  f(xk). 

Since  r^  >  0  and  r^  0  strictly,  s(rk)  >  s(rk+^  by  definition  of  s(r),  so 
k+1  k 

that  f(x  )  5  f(x  ).  This,  together  with  (4. 1),  also  implies  that 
I(xk+l)  2  I(xk). 

Since  we  also  have  f(x)  2  v*  for  x  e  S  f]  R,  we  must  have 

k  I  —  k 

f(x  )  4-  v  2  v*  .  V/e  know  that  at  least  one  limit  point  of  [x  }  exists,  and 

Theorem  3.  2  assures  us  that  limit  points  of  {xk}  must  be  in  A*,  so  that 

we  must  have  v  =  v*;  i.  e.  ,  f(xk)  v*  . 

The  conclusions  regarding  the  convergence  of  I(x  )  follow  from  the 

monotonicity  shown  above  and  from*  the  properties  (1)  and  (2)  of  I(x)  . 

Ey  property  (c)  of  the  U  function  and  by  the  result  for  U(x,  r) 

analagous  to  equation  (2.  1),  it  follows  that 


v* 


< 


k. 

lim.  U(x  \  r.  ) 
j-=°  k.' 


s  V*  +  s 


for  any  e>0  and  any  convergent  subsequence  [x  ^  ]  of  [x  J .  Hence, 

U(x  ,  r^  )  -v*.  It  follows  that  U(x  ,  r^)  ->  v*  ,  by  the  compactness  of 
R(^]  S  and  the  fact  that  at  least  one  convergent  subsequence  exists. 

From  the  above  results,  we  obtain  immediately  that  s(r I(x  )  -« 0  . 
Finally,  if  I(x)  £  0  in  R0/^!  S°  ,  then  it  follows  that 
U(x*  rk+l>  -  u(x»  rk)  f 
U(xk+I,  rk+1)  *  U(xk,  r 


or  r^  >  rk+i  >  0  and  any  x  crDs  ,  This  implies 


)  and  using  the  result  shown  above,  U(x 


k,  r.  )i  v*. 


C.  E.  D. 
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The  analagous  results  for  the  T(x,t)  function  can  be  proved  in  a 
similar  manner,  and  are  summarized  as  follows. 

Corollary  1  of  Theorem  3.  2. 

If  T(x,t)  =  f(x)  +  p(t)0(x),  where  O(x)  and  p(t)  satisfy  properties 
(1*)  -  (5*)  given  above,  f(x)  is  continuous,  {t^}  is  a  positive  strictly 
increasing  unbounded  sequence,  and  min^  T(x,  t^)  =  T(x  ,  tj^)  ,  then 
T(xk,tk)  |  v*  ,  f(xk)  t  v*  ,  0(xk)J/  0,  and  p(tk)0(xk)  -0. 


5.  Global  and  Dual  Results  for  Convex  Programming^ 

For  the  following  development,  we  utilize  the  formulation 
(C)  minimize  f(x)  subject  to  g^(x)  -  0,  i  =  1, .  . .  ,  m. 

Vre  shall  summarize  in  this  section  some  of  the  principal  results  that  hold 
when  the  interior  and  exterior  penalty  function  methods  are  applied  to  solve 
(C),  when  this  is  a  convex  programming  problem.  These  methods  are 
specialized  to  take  advantage  of  the  convexity  assumption,  essentially 
by  defining  the  penalty  functions  in  such  a  way  that  these  are  themse  ves 
convex  functions.  The  theorems  given  in  the  previous  section  are  of  course 
still  valid,  essentially  with  "local"  replaced  by  "global.  "  Furthermore, 
duality  results  are  forthcoming  that  provide  additional  information. 

We  first  discuss  the  interior  point  method.  It  is  assumed  that 
R°  =  {x  |g.(x)  >0,  i  =  l, . . .  ,m}^  (f.  V  e  shall  further  a  ssume  the  penalty 
factor  I  =  I  [g(x)J  ,  where  g  =  (g^, ...»  g^),  and  that  h(r)  =  r,  so  that  the 
form  of  the  interior  penalty  function  is  now  U(x,  r)  =  f(x)  +  rl  •  g(x)]  . 

Recall  that  I[g(x)]  is  continuous  in  F.°  and,  with  (xk)  C  R°  and 
xk-yc(F  -  R°)  ,  I  T g(xk) ]  -  +  ®  . 

It  is  possible  to  assure  the  convexity  of  U(x,  r)  in  RC  by 
appropriately  defining  I[g  (x)  ] .  The  following  lemma  is  immediate  and 
yields  a  suitable  definition. 

Lemma  5.1.  Convexity  of  l[g(x)]. 

If  1(g)  is  a  convex  dec-easing  function  of  g  for  g  >  0,  and  the 
g^(x)  are  concave,  then  I[g(x)]  is  a  convex  function  of  x  in  R°. 

^The  material  in  this  section  is  essentially  a  summary  of  some  of  the  results 
obtained  in  [5]  and  is  presented  here  for  completeness.  Jv'.ost  importantly, 
this  development  shows  how  the  penalty  function  can  be  used  to  take  advantage 
of  the  convexity  assumption  to  yield  global  and  dual  results. 


It  is  readily  verified,  for  exarrple,  that  the  functions  -Ef-ng. 
and  El/g^  satisfy  the  conditions  of  the  lemma.  Assuming  the  hypothesis 
of  the  lemn  a  holds,  then  I^g(x)]  is  convex,  and  hence, 

U(x,  r)  =  f(x)  +  rlTg(x)]  is  convex  in  R°,  for  r  >0  . 

The  following  theorem  then  holds  for  U(x,  r)  as  defined.  Note  that 
the  conclusions  are  immediately  implied  by  the  interior-point  auxiliary 
function  Theorem  3.  I,  and  Corollary  1  (Section  4)  of  that  theorem,  once  the 
convexity  assumptions  are  introduced. 

Theorem.  5.  1.  Interior  Penalty  h- .  ethod  Convergence  to  Solution  of  Convex 
Programming  Problem  f  5 3  - 

If  the  solutions  of  the  convex  programming  problem  (C )  are  bounded, 
R°  =  {x!g^(x)>0,  i  =  1, .  .  .  ,n--1  sf  0,  1(g)  is  defined,  decreasing  and 
convex  in  g  when  g  >0  and  (r^j  X  0,  then  U(x,r)  is  a  convex 
function  in  R°  for  r  >0,  for  r^  small  there  exists  x(r^)  that 
m  inimizes  U(x,  r^)  in  R°,  every  local  minimum,  of  U(x,  r^)  in 
R°  is  a  global  minimum  in  R°,  any  limit  point  x+  of  [x(r^)} 
solves  (C),  urx(rk),  rk]  J,  v*,  (f  rx(rk)]  }  \  v*  s  minRf(x),  and 
I  rx(rk)  ]  'v ,  where  y  =  +  <=  if  x*  €  (R  -  R°),  and  '"j  <  if 
x*eR°,  and  rk  lTx(rk)]  -  0  . 

As  expected  from  the  previous  general  theorem,  Theorem  3.  1,  and 
the  convexity  assumption,  the  minimizing  points  are  now  global  and  all  limit 
points  are  global  solutions  of  (C).  The  monotone  convergence  results  are 
also  global.  These  results  follow'  fron  the  convexity  of  U(x,  r)  and  the 
well  known  fact  that  any  local  solution  o’  the  convex  programming  problerr 
(C)  is  also  a  global  solution.  Hence,  the  compact  set  A*  of  local  minima 
of  (C)  is  the  unique  set  of  global  minima,  and  the  compact  set  S,  utilized 
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for  our  previous  local  results,  can  be  taken  to  be  any  compact  set  S  such 
that  S°OA*. 


In  effect,  the  role  of  convexity  is  essentially  to  make  the  previous 
theorem  a  "global"  one.  It  should  be  mentioned  that  in  [5]  a  condition 
is  given  that  regularizes  l[g  (x)]  such  that,  for  the  convex  problem  the 
minimizing  x(r^)  exists  for  all  r^  >  0,  not  merely  for  r^  small.  A 
similar  condition  has  not  been  obtained  for  the  general  problem. 

A  "dual"  of  problem  (C)  was  formulated  by  V'olfe  Til]  ,  assuming 
the  problem  functions  are  once  differentiable.  The  Lagrangian  function 


associated  with  problem  (C)  is  defined  as 
dual  of  (C)  is  then  defined  as, 


m 

L(x,u)  h  f(x)  -  T  u  g  (x). 

i=l  1  1 


The 


(D)  maximize  L(x,u) 

subject  to  v^L^x,  u)  =  0,  u,  SO,  i  =  1, . .  .  ,  m  . 

The  dual  relationship  is  defined  in  the  sense  given  in  the  following  two 
theorems. 

Theorem  5.2.  Primal- Dual  Bounds. 

If  (C)  is  a  convex  programming  problem,  y  is  any  primal  feasible 
point,  and  (x,u)  is  any  feasible  point  of  (D),  then  f(y)  2  L(x,u). 
Theorem  5.  3.  Equality  of  Primal- Dual  Values  at  Optimal  Solutions. 

If  (C)  is  a  convex  programming  problem  and  the  Kuhn- Tucker 
constraint  qualification  (see  Ref.  [6})  holds  at  a  solution  x*  of 
(C),  then  there  exists  a  solution  (x*,u*)  of  (D)  and  f(x*)  =  L(::*,u*)  . 
The  nondifferentiable  form  of  (D)  is  given  by  assuming  only  continuity 
and  replacing  V  L(x,u)  =  0  by  L(x,u)  =  inf  Lf^.u).  The  first  dual  theorem 
goes  through  easily  with  this  change.  By  making  this  substitution  and 


assuming  R°  s  {x  |g^(x)  >  0,  i  =  1, .  . .  ,  m}  }  <J)  ,  it  is  possible  to  dispense 


with  the  differentiability  requirement  and  the  Kuhn-Tucker  constraint 

qualification,  and  arrive  at  a  non- differentiable  version  of  the  second  dual 

theorem  given  above,  assuming  only  continuity. 

In  conjunction  with  the  Kuhn-Tucker  constraint  qualification,  it  i3 

relevant  to  point  out  the  fact  that  this  qualification  is  satisfied  providing 

the  constraints  are  differentiable  and  convex,  and  the  interior  of  the 

constraint  region  defined  by  the  nonlinear  constraints  is  nonempty  [13]. 

V  e  now  turn  briefly  to  the  significant  duality  results  that  follow  from 

the  U(x,  r)  method.  For  this  development,  we  give  U  additional  structure, 

m 

and  now  assume  that  the  penalty  factor  has  the  form  E  I.[g.(x)].  The 

i=l  1  1 

result  applies  to  the  nondifferentiable  form  of  the  dual  problem, 


(D)  maximize  L(x,  u)  subject  to  L(x,  u)  =  inf^  L(  P ,  u),  u.  2  0,  i  =  l, .  . .  ,  m. 


Theorem  5.4.  Dual- Feasibility  and  Convergence  for  Interior- Point  Methods  [5]. 

Assume  the  conditions  of  Theorem  5.1.  If,  in  addition,  each  L(g.) 

is  differentiable  in  g.  when  e.  >  0,  then  for  r.  small  enough 

l  °i  k 

rx(r,  ),u(r  )  ]  satisfies  the  constraints  of  (D)  where 

SIiMxM 

u.(r.  )  =  -r.  - - -  ,  i  =  1, . . .  ,m.  Furthermore, 

i  k  k  eg. 


lim^^  L[x(r^)  ,  u(r^)]  =  v*  =  min^f(x)  so  that  all  limit  points  of 
[rx(rk),  u(rk)]}  solve  (D). 

The  definition  of  the  quantities  u  (r  )  and  the  result  given  above 

i  k 

is  most  readily  appreciated  by  assuming  the  functions  differentiable, 
differentiating  U(x,  r)  directly,  and  comparing  the  result  to  V^Lfx.u),  which 


would  appear  in  the  differentiable  form  of  (D). 

The  above  result  is  a  significant  one,  not  generally  shared  by  other 
methods.  One  very  important  application  of  it  is  that  upper  and  lower 
bounds  on  the  optimal  value  v*  are  generated  with  the  determination  of 
each  minimizing  This  is  a  direct  consequence  of  the  first  dual 

theorem  given  above.  It  provides  a  natural  convergence  criterion.  Also, 
the  Lagrange  multipliers  are  critically  involved  in  providing  optimality 
conditions.  In  effect,  the  dual  theorem  provides  an  explicit  relationship 
between  U{x,  r)  and  the  Lagrangian  L(x, u)  associated  with  problem  (C). 

A  parallel  development  for  the  exterior- point  method  for  the  convex 
problem  is  given  in  [5].  Again,  similar  conditions  are  invoked  on  the 
form  of  T(x,t),  assuring  its  convexity.  The  convergence  theorem  is 
essentially  a  "global"  version  of  the  general  exterior- point  theorem  given 
in  Section  3.  Analagous  to  the  development  for  the  U(x,  r)  interior  method, 
a  condition  is  given  in  [5]  that  guarantees  the  existence  of  the  minimum  of 
T(x,t)  for  all  t  >0  ,  for  the  convex  problem.  Finally,  a  comparable  dual 
theorem  is  proved.  In  view  of  the  fact  that  the  optimal  value  is  approached 
from  below  in  this  method,  the  dual  result  provides  only  a  lower  bound  on 
the  optimal  value. 

Recalling  that  R°  ^  (j)  is  not  required  for  the  feasible- exterior 
T(x,t)  -  function  method,  the  above  indicated  results  imply,  in  particular, 
that  the  Kuhn- Tucker  constraint  qualification  need  not  be  satisfied  for 
problem  (C).  This  means,  when  conditions  assure  that  a  minimizing 
sequence  of  T(x,t^_)  leads  to  a  solution  of  the  problem,  then  we  have  a  way 
of  characterizing  such  a  solution  although  the  constraint  qualification  may 


not  hold. 
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